title	text	subject	date	fake	extracted_features	anonymized_text	num_entities	num_person_entities	num_org_entities	...	gpe_Georgia	gpe_Egypt	gpe_Benghazi	gpe_Italy	gpe_Sweden	gpe_Philippines	gpe_Oregon	gpe_Baghdad	gpe_Orlando	gpe_Illinois
15632	UPDATE: CATHOLIC UNIVERSITY THAT STATUE REMOVE...	This is a perfect example of the effort by lib...	politics	May 27, 2015	1	{'entities': [('American', 'NORP'), ('Indians'...	This is a perfect example of the effort by lib...	28	5	8	...	False	False	False	False	False	False	False	False	False	False
29490	Chelsea Manning criticizes Obama, draws Trump'...	WASHINGTON (Reuters) - Former U.S. military in...	politicsNews	January 26, 2017	0	{'entities': [('{RANDOM_ENTITY_GPE_0}', 'GPE')...	{RANDOM_ENTITY_GPE_0} ({RANDOM_ENTITY_ORG_1}) ...	51	23	6	...	False	False	False	False	False	False	False	False	False	False
22336	WAR ON WORDS: Facebook Censorship Widens, Webs...	21st Century Wire says The US media s neoMcCar...	US_News	December 3, 2016	1	{'entities': [('{RANDOM_ENTITY_GPE_0}', 'GPE')...	21st Century Wire says The {RANDOM_ENTITY_GPE_...	71	3	37	...	False	False	False	False	False	False	False	False	False	False
19487	DEMOCRATS AREN’T AFRAID Trump Will Be A Terrib...	The brilliant, outspoken and beautiful Tomi La...	left-news	Dec 7, 2016	1	{'entities': [('{RANDOM_ENTITY_PERSON_0}', 'PE...	The brilliant, outspoken and beautiful {RANDOM...	1	1	0	...	False	False	False	False	False	False	False	False	False	False
43007	Palestinian President Abbas says peace closer ...	NEW YORK (Reuters) - Palestinian President Mah...	worldnews	September 20, 2017	0	{'entities': [('{RANDOM_ENTITY_GPE_0}', 'GPE')...	{RANDOM_ENTITY_GPE_0} ({RANDOM_ENTITY_ORG_1}) ...	87	21	7	...	False	False	False	False	False	False	False	False	False	False
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
15714	HAPPY MOTHER’S DAY…Your First Grader Just Rate...	Rate your mom for Mother s Day sounds like the...	politics	May 11, 2015	1	{'entities': [('first', 'ORDINAL'), ('Ridgecre...	Rate your mom for {RANDOM_ENTITY_DATE_2} sound...	16	1	1	...	False	False	False	False	False	False	False	False	False	False
3926	Trump Goes COMPLETELY Off The Rails Over Hill...	The right wing is still harping on the FBI s d...	News	November 5, 2016	1	{'entities': [('{RANDOM_ENTITY_ORG_0}', 'ORG')...	The right wing is still harping on the {RANDOM...	23	13	4	...	False	False	False	False	False	False	False	False	False	False
35350	Chile election ends era of female presidents i...	SANTIAGO (Reuters) - When Chile s President Mi...	worldnews	December 19, 2017	0	{'entities': [('{RANDOM_ENTITY_ORG_0}', 'ORG')...	{RANDOM_ENTITY_ORG_0} ({RANDOM_ENTITY_ORG_1}) ...	101	29	11	...	False	False	False	False	False	False	False	False	False	False
3236	Charlie Sheen Sends SHOCKING Message About Tr...	This year was a rough one and many of us can t...	News	December 29, 2016	1	{'entities': [('{RANDOM_ENTITY_DATE_0}', 'DATE...	{RANDOM_ENTITY_DATE_0} was a rough one and man...	40	26	2	...	False	False	False	False	False	False	False	False	False	False
38093	Venezuela's ex-prosecutor wants Maduro tried a...	CARACAS (Reuters) - Venezuela s sacked former ...	worldnews	November 16, 2017	0	{'entities': [('{RANDOM_ENTITY_ORG_0}', 'ORG')...	CARACAS ({RANDOM_ENTITY_ORG_0}) - {RANDOM_ENTI...	37	14	5	...	False	False	False	False	False	False	False	False	False	False

200 rows × 289 columns

289

	Column	Type
0	title	object
1	text	object
2	subject	object
3	date	object
4	fake	int64
5	extracted_features	object
6	anonymized_text	object
7	num_entities	int64
8	num_person_entities	int64
9	num_org_entities	int64
10	num_gpe_entities	int64
11	num_date_entities	int64
12	sentiment	float64
13	num_words	int64
14	num_sents	int64
15	avg_word_length	float64
16	lexical_diversity	float64
17	readability	float64
18	num_tokens	int64
19	char_count	int64
20	ner_density	float64
21	max_dep_depth	int64
22	avg_token_vector_norm	float64
23	avg_sent_length	float64
24	median_sent_length	float64
25	std_sent_length	float64
26	punctuation_density	float64
27	stop_words_ratio	float64
28	pos_ratio_PROPN	float64
29	pos_ratio_AUX	float64
30	pos_ratio_ADP	float64
31	pos_ratio_INTJ	float64
32	pos_ratio_PART	float64
33	pos_ratio_VERB	float64
34	pos_ratio_ADV	float64
35	pos_ratio_SYM	float64
36	pos_ratio_NOUN	float64
37	pos_ratio_ADJ	float64
38	pos_ratio_NUM	float64
39	pos_ratio_PRON	float64
40	pos_ratio_SPACE	float64
41	pos_ratio_X	float64
42	pos_ratio_SCONJ	float64
43	pos_ratio_DET	float64
44	pos_ratio_PUNCT	float64
45	pos_ratio_CCONJ	float64
46	verb_tense_ratio_past	float64
47	verb_tense_ratio_present	int64
48	verb_tense_ratio_future	int64
49	person_Trump	bool
50	person_Obama	bool
51	person_Clinton	bool
52	person_Hillary	bool
53	person_Sanders	bool
54	person_Cruz	bool
55	person_Putin	bool
56	person_Ryan	bool
57	person_Bush	bool
58	person_Flynn	bool
59	person_@realDonaldTrump	bool
60	person_McCain	bool
61	person_Obamacare	bool
62	person_Rubio	bool
63	person_Merkel	bool
64	person_Donald Trump s	bool
65	person_James Comey	bool
66	person_Tillerson	bool
67	person_Mike Pence	bool
68	person_Johnson	bool
69	person_McConnell	bool
70	person_Moore	bool
71	person_Kelly	bool
72	person_Assad	bool
73	person_Carson	bool
74	person_Hillary Clinton s	bool
75	person_Bannon	bool
76	person_Warren	bool
77	person_Donald	bool
78	person_Hariri	bool
79	person_Sean Spicer	bool
80	person_Mitt Romney	bool
81	person_Erdogan	bool
82	person_Jones	bool
83	person_Conway	bool
84	person_Biden	bool
85	person_Mattis	bool
86	person_Macron	bool
87	person_Graham	bool
88	person_Brown	bool
89	person_Xi	bool
90	person_Schumer	bool
91	person_Kushner	bool
92	person_Mnuchin	bool
93	person_Scalia	bool
94	person_Robert Mueller	bool
95	person_Soros	bool
96	person_Manafort	bool
97	person_Reagan	bool
98	person_Palin	bool
99	person_Mugabe	bool
100	person_Ivanka	bool
101	person_Kasich	bool
102	person_Gorsuch	bool
103	person_Don	bool
104	person_Jeff Sessions	bool
105	person_Kennedy	bool
106	person_Netanyahu	bool
107	person_Miller	bool
108	person_Abe	bool
109	person_Khan	bool
110	person_Reilly	bool
111	person_Stein	bool
112	person_Williams	bool
113	person_Reid	bool
114	person_Haley	bool
115	person_Smith	bool
116	person_Rice	bool
117	person_Nancy Pelosi	bool
118	person_Paul	bool
119	person_Ross	bool
120	person_Lee	bool
121	person_Weinstein	bool
122	person_Davis	bool
123	person_Bill	bool
124	person_Breitbart	bool
125	person_Carter	bool
126	person_King	bool
127	person_John Kerry	bool
128	person_Garland	bool
129	org_Reuters	bool
130	org_the White House	bool
131	org_Senate	bool
132	org_Congress	bool
133	org_FBI	bool
134	org_State	bool
135	org_EU	bool
136	org_CNN	bool
137	org_the State Department	bool
138	org_the Supreme Court	bool
139	org_GOP	bool
140	org_Fox News	bool
141	org_Facebook	bool
142	org_U.N.	bool
143	org_the New York Times	bool
144	org_Twitter	bool
145	org_CIA	bool
146	org_the Washington Post	bool
147	org_the United Nations	bool
148	org_ISIS	bool
149	org_the House of Representatives	bool
150	org_NATO	bool
151	org_the European Union	bool
152	org_the Republican Party	bool
153	org_the Democratic Party	bool
154	org_DNC	bool
155	org_Commission	bool
156	org_Comey	bool
157	org_NBC	bool
158	org_Pentagon	bool
159	org_MSNBC	bool
160	org_Army	bool
161	org_Fox	bool
162	org_Kremlin	bool
163	org_EPA	bool
164	org_Treasury	bool
165	org_the Department of Justice	bool
166	org_Breitbart News	bool
167	org_the U.N. Security Council	bool
168	org_Planned Parenthood	bool
169	org_the Clinton Foundation	bool
170	org_Medicaid	bool
171	org_Brexit	bool
172	org_al Qaeda	bool
173	org_Hezbollah	bool
174	org_ABC News	bool
175	org_Islam	bool
176	org_Politico	bool
177	org_Defense	bool
178	org_Capitol Hill	bool
179	org_Google	bool
180	org_Fed	bool
181	org_NFL	bool
182	org_Navy	bool
183	org_Group	bool
184	org_NSA	bool
185	org_Taliban	bool
186	org_ABC	bool
187	org_CBS	bool
188	org_WikiLeaks	bool
189	org_SPD	bool
190	org_the Democratic National Committee	bool
191	org_Mueller	bool
192	org_Justice Department	bool
193	org_the Wall Street Journal	bool
194	org_DOJ	bool
195	org_Daily Mail	bool
196	org_the Senate Intelligence Committee	bool
197	org_Getty Images	bool
198	org_NRA	bool
199	org_MOSCOW	bool
200	org_DHS	bool
201	org_the Department of Homeland Security	bool
202	org_LePage	bool
203	org_Confederate	bool
204	org_the Senate Judiciary Committee	bool
205	org_RNC	bool
206	org_Cabinet	bool
207	org_IRS	bool
208	org_Secret Service	bool
209	gpe_U.S.	bool
210	gpe_the United States	bool
211	gpe_Washington	bool
212	gpe_Russia	bool
213	gpe_America	bool
214	gpe_China	bool
215	gpe_Trump	bool
216	gpe_North Korea	bool
217	gpe_Iran	bool
218	gpe_Syria	bool
219	gpe_New York	bool
220	gpe_Iraq	bool
221	gpe_Mexico	bool
222	gpe_Britain	bool
223	gpe_Israel	bool
224	gpe_Texas	bool
225	gpe_Turkey	bool
226	gpe_Florida	bool
227	gpe_Germany	bool
228	gpe_Moscow	bool
229	gpe_Myanmar	bool
230	gpe_California	bool
231	gpe_Saudi Arabia	bool
232	gpe_London	bool
233	gpe_Beijing	bool
234	gpe_North Carolina	bool
235	gpe_Japan	bool
236	gpe_France	bool
237	gpe_Afghanistan	bool
238	gpe_Paris	bool
239	gpe_Virginia	bool
240	gpe_India	bool
241	gpe_Jerusalem	bool
242	gpe_Ohio	bool
243	gpe_Cuba	bool
244	gpe_Ukraine	bool
245	gpe_Yemen	bool
246	gpe_Libya	bool
247	gpe_Chicago	bool
248	gpe_Brussels	bool
249	gpe_Pyongyang	bool
250	gpe_Canada	bool
251	gpe_Australia	bool
252	gpe_Michigan	bool
253	gpe_Pakistan	bool
254	gpe_Puerto Rico	bool
255	gpe_Iowa	bool
256	gpe_Lebanon	bool
257	gpe_Bangladesh	bool
258	gpe_Arizona	bool
259	gpe_Taiwan	bool
260	gpe_Hollywood	bool
261	gpe_Spain	bool
262	gpe_UK	bool
263	gpe_Alabama	bool
264	gpe_Tehran	bool
265	gpe_Berlin	bool
266	gpe_New Jersey	bool
267	gpe_Wisconsin	bool
268	gpe_New York City	bool
269	gpe_D.C.	bool
270	gpe_Venezuela	bool
271	gpe_Kansas	bool
272	gpe_Pennsylvania	bool
273	gpe_Vietnam	bool
274	gpe_Poland	bool
275	gpe_Flint	bool
276	gpe_New Hampshire	bool
277	gpe_Catalonia	bool
278	gpe_Ireland	bool
279	gpe_Georgia	bool
280	gpe_Egypt	bool
281	gpe_Benghazi	bool
282	gpe_Italy	bool
283	gpe_Sweden	bool
284	gpe_Philippines	bool
285	gpe_Oregon	bool
286	gpe_Baghdad	bool
287	gpe_Orlando	bool
288	gpe_Illinois	bool

	Unnamed: 0	title	text	subject	date	fake	extracted_features	anonymized_text	num_entities	num_person_entities	...	gpe_Pyongyang	gpe_Canada	gpe_Australia	gpe_Michigan	gpe_Pakistan	gpe_Puerto Rico	gpe_Iowa	gpe_Lebanon	gpe_Bangladesh	gpe_Arizona
11	12038	WHERE’S HILLARY? CLINTON SPOTTED Dining Alone		politics	Dec 30, 2016	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
41	12750	WHOA! RUSH LIMBAUGH RIPS Into Republicans Who ...		politics	Oct 12, 2016	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
177	13008	HILLARY TAKES CREDIT For The Arab Spring Disas...		politics	Sep 15, 2016	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
324	11478	CHILLING! FOX REPORTER JAMES ROSEN Recounts Be...		politics	Mar 6, 2017	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
400	11854	LIVE FEED: INAUGURATION 2017!		politics	Jan 20, 2017	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
44632	12183	TUCKER CARLSON DESTROYS Smug Elector Who Refus...	https://youtu.be/7oOhwHG2Gb4	politics	Dec 9, 2016	1	{'entities': [], 'entity_counts': {}, 'person_...	https://youtu.be/7oOhwHG2Gb4	0	0	...	False	False	False	False	False	False	False	False	False	False
44782	19115	TUCKER CARLSON Defends Trump On Sweden Comment...		left-news	Feb 20, 2017	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
44868	19118	AWESOME! HISPANIC TRUMP SUPPORTER Rips Into Pr...		left-news	Feb 19, 2017	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
44878	19769	DEMOCRAT OPERATIVES Caught Planning To Bully W...		left-news	Oct 20, 2016	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
44893	11284	UNREAL! CBS’S TED KOPPEL Tells Sean Hannity He...		politics	Mar 27, 2017	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False

836 rows × 169 columns

<Axes: title={'center': 'Text Size (Spacy Token Count)'}, xlabel='Text Length', ylabel='Density'>

1.4 Entity Analysis

	col	total	fake	true	prop_fake
0	person_Trump	21541	11981	9560	0.556195
80	org_Reuters	21531	302	21229	0.014026
15	person_Donald Trump s	18051	8873	9178	0.491552
10	person_@realDonaldTrump	16995	7961	9034	0.468432
160	gpe_U.S.	16027	4671	11356	0.291446
...	...	...	...	...	...
61	person_Reilly	217	206	11	0.949309
53	person_Gorsuch	217	79	138	0.364055
226	gpe_Flint	201	145	56	0.721393
50	person_Mugabe	192	4	188	0.020833
140	org_SPD	179	6	173	0.033520

240 rows × 5 columns

The importance score combines two factors: how often an entity appears (frequency) and how strongly it's associated with fake news (predictive power). It's calculated by normalizing and averaging these two measures. A higher score indicates that an entity is both common and a good predictor of fake news. This helps identify the most relevant entities for distinguishing between fake and real news articles.

	total	prop_fake	importance_score
total	1.000000	-0.063512	0.388932
prop_fake	-0.063512	1.000000	-0.277533
importance_score	0.388932	-0.277533	1.000000

1.4.1 PCA

	fake	num_entities	num_person_entities	num_org_entities	num_gpe_entities	num_date_entities	sentiment	num_words	num_sents	avg_word_length	...	pos_ratio_PRON	pos_ratio_SPACE	pos_ratio_X	pos_ratio_SCONJ	pos_ratio_DET	pos_ratio_PUNCT	pos_ratio_CCONJ	verb_tense_ratio_past	verb_tense_ratio_present	verb_tense_ratio_future
22216	1	13	6	4	3	0	0.08333	187	7	4.58824	...	0.01070	0.02674	0.00000	0.01070	0.12299	0.05348	0.04278	0.45455	0	0
27917	0	114	49	28	18	10	0.08660	904	29	4.47898	...	0.04978	0.00221	0.00000	0.00664	0.08407	0.11394	0.02102	0.60759	0	0
25007	0	26	5	4	6	4	-0.00504	359	14	4.30084	...	0.06128	0.01114	0.00000	0.01114	0.08635	0.12813	0.01393	0.64706	0	0
1377	1	24	8	1	8	3	-0.01116	230	12	4.61739	...	0.08696	0.06957	0.00000	0.01739	0.07391	0.10870	0.01304	0.73077	0	0
32476	0	75	10	15	19	9	0.03935	606	23	4.30363	...	0.06601	0.00165	0.00000	0.01320	0.08581	0.10561	0.01980	0.53846	0	0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
11284	1	0	0	0	0	0	0.00000	1	1	1.00000	...	0.00000	1.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0	0
44732	0	121	17	20	34	22	0.06335	807	24	4.50310	...	0.03346	0.01735	0.00000	0.01611	0.06568	0.08674	0.02726	0.52830	0	0
38158	0	6	1	1	1	2	-0.10606	89	5	4.02247	...	0.11236	0.04494	0.00000	0.01124	0.06742	0.13483	0.02247	0.28571	0	0
860	1	111	26	21	1	25	-0.03391	1092	48	3.83700	...	0.06868	0.04579	0.00183	0.02198	0.05586	0.14011	0.01282	0.28986	0	0
15795	1	36	10	13	4	3	0.03011	290	10	4.21724	...	0.03103	0.01034	0.00000	0.00690	0.10000	0.10345	0.01379	0.57143	0	0

44898 rows × 43 columns

Number of components needed to explain 95% of the variance: 25

t-SNE Results Shape: (44898, 2)
t-SNE Range - X: -94.89072 to 95.122025
t-SNE Range - Y: -95.44623 to 91.66013

/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.
A parameter name that contains `beta` will be renamed internally to `bias`. Please use a different name to suppress this warning.
A parameter name that contains `gamma` will be renamed internally to `weight`. Please use a different name to suppress this warning.

Generating embeddings:   3%|▎         | 6/176 [00:27<13:13,  4.67s/it]

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[67], line 3
      1 embedding_df = ds.full_data.copy()
----> 3 embeddings_tinybert = get_embeddings(embedding_df["text"], model,  tokenizer, batch_size=256, use_gpu=True, model_type='auto')

Cell In[62], line 54, in get_embeddings(texts, model, tokenizer, batch_size, max_length, use_gpu, model_type)
     52             outputs = model(**batch)
     53             batch_embeddings = outputs.last_hidden_state[:, 0, :]
---> 54             embeddings.append(batch_embeddings.cpu())
     56     return torch.cat(embeddings).numpy()
     58 elif model_type == 'sentence_transformer':

KeyboardInterrupt: 

Generating embeddings: 100%|██████████| 176/176 [04:42<00:00,  1.61s/it]

array([[-0.04292417,  0.04017529,  0.00822352, ...,  0.00830901,
         0.04013212, -0.05353491],
       [-0.0300238 , -0.05997578,  0.09334268, ..., -0.08667795,
        -0.02973704,  0.06296595],
       [ 0.0077344 ,  0.01727735,  0.01518998, ..., -0.06386147,
        -0.015207  , -0.02204838],
       ...,
       [-0.00774679, -0.00621239,  0.02480308, ..., -0.05097079,
         0.00514519,  0.00038478],
       [ 0.06024081,  0.06292548,  0.00465201, ..., -0.00634774,
         0.04170959,  0.02873757],
       [ 0.00446899, -0.08325394, -0.00282552, ..., -0.16707481,
         0.09308673, -0.04893658]], dtype=float32)

1.5 Temporal Analysis

<Figure size 1900x1100 with 0 Axes>

We can see that the dataset is not at all balanced {time wise} which is highly problematic because {}

	Unnamed: 0	title	text	subject	date	fake	extracted_features	anonymized_text	num_entities	num_person_entities	...	gpe_Pyongyang	gpe_Canada	gpe_Australia	gpe_Michigan	gpe_Pakistan	gpe_Puerto Rico	gpe_Iowa	gpe_Lebanon	gpe_Bangladesh	gpe_Arizona
0	22216	Ben Stein Calls Out 9th Circuit Court: Committ...	21st Century Wire says Ben Stein, reputable pr...	US_News	February 13, 2017	1	{'entities': [('{RANDOM_ENTITY_PERSON_0}', 'PE...	21st Century Wire says {RANDOM_ENTITY_PERSON_0...	13	6	...	False	False	False	False	False	False	False	False	False	False
1	27917	Trump drops Steve Bannon from National Securit...	WASHINGTON (Reuters) - U.S. President Donald T...	politicsNews	April 5, 2017	0	{'entities': [('{RANDOM_ENTITY_GPE_0}', 'GPE')...	{RANDOM_ENTITY_GPE_0} ({RANDOM_ENTITY_ORG_1}) ...	114	49	...	False	False	False	False	False	False	False	False	False	False
2	25007	Puerto Rico expects U.S. to lift Jones Act shi...	(Reuters) - Puerto Rico Governor Ricardo Rosse...	politicsNews	September 27, 2017	0	{'entities': [('{RANDOM_ENTITY_ORG_0}', 'ORG')...	({RANDOM_ENTITY_ORG_0}) - {RANDOM_ENTITY_GPE_1...	26	5	...	False	False	False	False	False	True	False	False	False	False
3	1377	OOPS: Trump Just Accidentally Confirmed He Le...	On Monday, Donald Trump once again embarrassed...	News	May 22, 2017	1	{'entities': [('{RANDOM_ENTITY_DATE_0}', 'DATE...	On {RANDOM_ENTITY_DATE_0}, {RANDOM_ENTITY_PERS...	24	8	...	False	False	False	False	False	False	False	False	False	False
4	32476	Donald Trump heads for Scotland to reopen a go...	GLASGOW, Scotland (Reuters) - Most U.S. presid...	politicsNews	June 24, 2016	0	{'entities': [('{RANDOM_ENTITY_GPE_0}', 'GPE')...	GLASGOW, {RANDOM_ENTITY_GPE_0} ({RANDOM_ENTITY...	75	10	...	False	False	False	False	False	False	False	False	False	False
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
44893	11284	UNREAL! CBS’S TED KOPPEL Tells Sean Hannity He...		politics	Mar 27, 2017	1	{'entities': [], 'entity_counts': {}, 'person_...		0	0	...	False	False	False	False	False	False	False	False	False	False
44894	44732	PM May seeks to ease Japan's Brexit fears duri...	LONDON/TOKYO (Reuters) - British Prime Ministe...	worldnews	August 29, 2017	0	{'entities': [('{RANDOM_ENTITY_GPE_0}', 'GPE')...	{RANDOM_ENTITY_GPE_0}/TOKYO ({RANDOM_ENTITY_OR...	121	17	...	False	False	False	False	False	False	False	False	False	False
44895	38158	Merkel: Difficult German coalition talks can r...	BERLIN (Reuters) - Chancellor Angela Merkel sa...	worldnews	November 16, 2017	0	{'entities': [('{RANDOM_ENTITY_GPE_0}', 'GPE')...	{RANDOM_ENTITY_GPE_0} ({RANDOM_ENTITY_ORG_1}) ...	6	1	...	False	False	False	False	False	False	False	False	False	False
44896	860	Trump Stole An Idea From North Korean Propaga...	Jesus f*cking Christ our President* is a moron...	News	July 14, 2017	1	{'entities': [('{RANDOM_ENTITY_PERSON_0}', 'PE...	{RANDOM_ENTITY_PERSON_0} f*cking Christ our Pr...	111	26	...	False	False	False	False	False	False	False	False	False	False
44897	15795	BREAKING: HILLARY CLINTON’S STATE DEPARTMENT G...	IF SHE S NOT TOAST NOW THEN WE RE IN BIGGER TR...	politics	Apr 23, 2015	1	{'entities': [('{RANDOM_ENTITY_PERSON_0}', 'PE...	IF SHE S NOT TOAST NOW THEN WE RE IN BIGGER TR...	36	10	...	False	False	False	False	False	False	False	False	False	False

44898 rows × 200 columns

3.0 Heuristics Based Model

Our goal is not to maximize performance but rather to {find issues in the dataset}:

• Is the model overfitting on some entitites? e.g. Reuters might almost never report "fake" news, but we can't guarantee that won't change in the future. Also, a model that would rely on specific keywords would be trivial to game.

3.1 Feature Selection

We will start by dropping all entity features that have > 0.25 normalized importance or higher, this includes these features:

Dropping total columns: 90

	col	total	fake	true	prop_fake	total_normalized	prop_fake_normalized	importance_score
80	org_Reuters	21531	302	21229	0.014026	0.479554	-0.508959	0.494256
162	gpe_Washington	10768	1973	8795	0.183228	0.239833	-0.339757	0.289795
164	gpe_America	6650	5505	1145	0.827820	0.148114	0.304834	0.226474
3	person_Hillary	2943	2764	179	0.939178	0.065549	0.416192	0.240870
165	gpe_China	2611	492	2119	0.188434	0.058154	-0.334552	0.196353
...	...	...	...	...	...	...	...	...
154	org_Confederate	249	201	48	0.807229	0.005546	0.284243	0.144895
158	org_IRS	234	195	39	0.833333	0.005212	0.310348	0.157780
61	person_Reilly	217	206	11	0.949309	0.004833	0.426323	0.215578
50	person_Mugabe	192	4	188	0.020833	0.004276	-0.502152	0.253214
140	org_SPD	179	6	173	0.033520	0.003987	-0.489466	0.246726

90 rows × 8 columns

282
192

	number	value	datetime_start	datetime_complete	duration	params_bagging_fraction	params_bagging_freq	params_feature_fraction	params_lambda_l1	params_lambda_l2	params_learning_rate	params_min_child_samples	params_num_leaves	state
1	1	0.999845	2024-08-13 18:51:21.834861	2024-08-13 18:51:25.769572	0 days 00:00:03.934711	0.972919	6	0.118526	4.329370e-07	4.473429e-07	0.133112	25	159	COMPLETE
2	2	0.999713	2024-08-13 18:51:25.772550	2024-08-13 18:51:29.750344	0 days 00:00:03.977794	0.362106	5	0.488751	4.258943e-06	1.982698e-05	0.037521	18	88	COMPLETE
0	0	0.999662	2024-08-13 18:51:19.022850	2024-08-13 18:51:21.831381	0 days 00:00:02.808531	0.638793	2	0.758795	3.332365e-08	6.245760e-01	0.711448	19	105	COMPLETE

Training until validation scores don't improve for 50 rounds
[100]	valid_0's auc: 0.9976
[200]	valid_0's auc: 0.997842
[300]	valid_0's auc: 0.997924
[400]	valid_0's auc: 0.99803
Early stopping, best iteration is:
[402]	valid_0's auc: 0.998034
Training LightGBM model took 7.13 seconds

Test Metrics:

{'accuracy': 0.9816258351893096,
 'precision': 0.9840255591054313,
 'recall': 0.9808917197452229,
 'f1': 0.9824561403508771,
 'auc': 0.998033632164362,
 'log_loss': 0.08163107432287664}

/usr/local/lib/python3.10/dist-packages/shap/explainers/_tree.py:448: UserWarning: LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray
  warnings.warn('LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray')

/usr/local/lib/python3.10/dist-packages/shap/explainers/_tree.py:448: UserWarning: LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray
  warnings.warn('LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray')
/usr/local/lib/python3.10/dist-packages/shap/explainers/_tree.py:448: UserWarning: LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray
  warnings.warn('LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray')

	feature	importance	importance_percentage	mean_abs_shap	shap_importance_percentage
5	pos_ratio_PUNCT	1259	3.035344	4.402535	10.296663
9	ner_density	1204	2.902744	2.211655	5.172626
2	verb_tense_ratio_past	1383	3.334298	2.019577	4.723392
3	avg_token_vector_norm	1364	3.288490	1.865964	4.364123
16	punctuation_density	1123	2.707459	1.830244	4.280579
1	std_sent_length	1397	3.368051	1.689886	3.952311
8	pos_ratio_ADV	1246	3.004002	1.572365	3.677453
30	num_date_entities	746	1.798544	1.551266	3.628105
25	num_gpe_entities	884	2.131250	1.450836	3.393219
13	avg_word_length	1150	2.772554	1.380218	3.228058
23	avg_sent_length	983	2.369931	1.142517	2.672123
15	pos_ratio_PRON	1130	2.724336	1.127723	2.637522
4	pos_ratio_SPACE	1266	3.052220	1.093518	2.557523
26	median_sent_length	881	2.124018	1.081556	2.529546
22	pos_ratio_PROPN	1008	2.430204	1.071308	2.505578
41	gpe_U.S.	73	0.175997	1.050933	2.457926
28	num_person_entities	835	2.013115	0.928912	2.172543
6	pos_ratio_PART	1252	3.018468	0.849034	1.985723
40	person_Obama	86	0.207339	0.798815	1.868271
43	person_Trump	54	0.130189	0.781697	1.828234
29	num_org_entities	802	1.933555	0.725261	1.696243
32	num_words	585	1.410386	0.652742	1.526636
24	pos_ratio_NUM	896	2.160181	0.648339	1.516337
56	person_@realDonaldTrump	41	0.098848	0.598985	1.400909
12	pos_ratio_SCONJ	1173	2.828005	0.575131	1.345118

/usr/local/lib/python3.10/dist-packages/shap/explainers/_tree.py:448: UserWarning: LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray
  warnings.warn('LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray')

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[34], line 2
      1 # Plot SHAP dependence for a specific feature
----> 2 shap.plot_shap_dependence(final_model_no_r, X_test_no_r, "pos_ratio_PUNCT")

File /mnt/v/projects/DL_s4/src/models/shared/shap_analysis.py:67, in plot_shap_dependence(model, X, feature)
     64 shap_values = explainer.shap_values(X)
     66 plt.figure(figsize=(10, 6))
---> 67 shap.dependence_plot(feature, shap_values[1], X, show=False)
     68 plt.title(f"SHAP Dependence Plot for {feature}")
     69 plt.tight_layout()

File /usr/local/lib/python3.10/dist-packages/shap/plots/_scatter.py:572, in dependence_legacy(ind, shap_values, features, feature_names, display_features, interaction_index, color, axis_color, cmap, dot_size, x_jitter, alpha, title, xmin, xmax, ax, show, ymin, ymax)
    570 if not hasattr(ind, "__len__"):
    571     if interaction_index == "auto":
--> 572         interaction_index = approximate_interactions(ind, shap_values, features)[0]
    573     interaction_index = convert_name(interaction_index, shap_values, feature_names)
    574 categorical_interaction = False

File /usr/local/lib/python3.10/dist-packages/shap/utils/_general.py:123, in approximate_interactions(index, shap_values, X, feature_names)
    121 x = X[inds, index]
    122 srt = np.argsort(x)
--> 123 shap_ref = shap_values[inds, index]
    124 shap_ref = shap_ref[srt]
    125 inc = max(min(int(len(x) / 10.0), 50), 1)

IndexError: index 37 is out of bounds for axis 1 with size 1

<Figure size 1000x600 with 0 Axes>

3.3 Feature Analysis

3.3.1 pos_ratio_PUNCT

'pos_ratio_PUNCT' is the ratio of punctuation tokens to the total number of tokens (based on Spacy 'PUNCT' tag). This feature indicates the relative frequency of punctuation in the text, which can be informative about writing style, formality, or the type of content.

3.3.2 ner_DENSITY

'ner_density' represents the ratio of named entities to the total number of tokens in a text. It's calculated by dividing the count of named entities (like persons, organizations, locations) by the total number of tokens. So basically a higher ner_density indicates that a larger proportion of the text consists of named entities, interestingly "fake" news articles have a generally higher proportion of named entities to other text.

3.3.3 verb_tense_ratio_past

The proportion of verbs in the text that are in the past tense. A higher value indicates more historical or narrative content.

3.3.4 avg_token_vector_norm

The average magnitude of word vectors in the text. Higher values suggest more semantically rich or specific vocabulary.

3.3.5 punctuation_density

The ratio of punctuation marks to total characters. Higher values may indicate more complex sentence structures or dialogue-heavy text

3.3.6 std_sent_length

Standard deviation of sentence lengths. Higher values suggest more variability in sentence structure, potentially indicating more complex writing.

3.3.7 pos_ratio_ADV

The proportion of adverbs in the text. A higher ratio might indicate more descriptive or nuanced language, often used to modify verbs, adjectives, or other adverbs

['person_Trump',
 'person_Obama',
 'person_Clinton',
 'person_Hillary',
 'person_Sanders',
 'person_Cruz',
 'person_Putin',
 'person_Ryan',
 'person_Bush',
 'person_McCain',
 'person_@realDonaldTrump',
 'person_Rubio',
 'person_Donald Trump s',
 'person_Flynn',
 'person_Obamacare',
 'person_Comey',
 'person_Merkel',
 'person_Pence',
 'person_Moore',
 'person_McConnell',
 'person_Tillerson',
 'person_Johnson',
 'person_Kelly',
 'person_Assad',
 'person_Carson',
 'person_Hillary Clinton s',
 'person_Bannon',
 'person_Warren',
 'person_Donald',
 'person_Conway',
 'person_Jones',
 'person_Sean Spicer',
 'person_Hariri',
 'person_Mitt Romney',
 'person_Erdogan',
 'person_Biden',
 'person_Brown',
 'person_Xi',
 'person_Graham',
 'person_Scalia',
 'person_Schumer',
 'person_Palin',
 'person_Soros',
 'person_Mattis',
 'person_Kushner',
 'person_Mnuchin',
 'person_Manafort',
 'person_Macron',
 'person_Robert Mueller',
 'person_Reagan',
 'org_the White House',
 'org_Senate',
 'org_Congress',
 'org_FBI',
 'org_State',
 'org_CNN',
 'org_EU',
 'org_GOP',
 'org_the State Department',
 'org_the Supreme Court',
 'org_Fox News',
 'org_Facebook',
 'org_the New York Times',
 'org_U.N.',
 'org_the Washington Post',
 'org_CIA',
 'org_ISIS',
 'org_Twitter',
 'org_the Republican Party',
 'org_NATO',
 'org_the Democratic Party',
 'org_the United Nations',
 'org_DNC',
 'org_Comey',
 'org_the House of Representatives',
 'org_the European Union',
 'org_Commission',
 'org_MSNBC',
 'org_NBC',
 'org_Army',
 'org_Justice Department',
 'org_Pentagon',
 'org_Fox',
 'org_EPA',
 'org_Breitbart News',
 'org_the Clinton Foundation',
 'org_Planned Parenthood',
 'org_U.N. Security Council',
 'org_Treasury',
 'org_the Department of Justice',
 'org_al Qaeda',
 'org_Islam',
 'org_Hezbollah',
 'org_Kremlin',
 'org_Medicaid',
 'org_ABC News',
 'org_Capitol Hill',
 'org_Politico',
 'org_NFL',
 'org_Google',
 'gpe_U.S.',
 'gpe_the United States',
 'gpe_Russia',
 'gpe_America',
 'gpe_Trump',
 'gpe_China',
 'gpe_Washington',
 'gpe_Iran',
 'gpe_North Korea',
 'gpe_Syria',
 'gpe_New York',
 'gpe_Iraq',
 'gpe_Mexico',
 'gpe_Israel',
 'gpe_Britain',
 'gpe_Texas',
 'gpe_Florida',
 'gpe_Germany',
 'gpe_Turkey',
 'gpe_California',
 'gpe_Moscow',
 'gpe_Myanmar',
 'gpe_Saudi Arabia',
 'gpe_North Carolina',
 'gpe_Japan',
 'gpe_France',
 'gpe_Virginia',
 'gpe_India',
 'gpe_Afghanistan',
 'gpe_Cuba',
 'gpe_Ohio',
 'gpe_Beijing',
 'gpe_London',
 'gpe_Ukraine',
 'gpe_Jerusalem',
 'gpe_Paris',
 'gpe_Michigan',
 'gpe_D.C.',
 'gpe_Yemen',
 'gpe_Chicago',
 'gpe_Libya',
 'gpe_Canada',
 'gpe_Australia',
 'gpe_Iowa',
 'gpe_Pakistan',
 'gpe_Pyongyang',
 'gpe_Arizona',
 'gpe_Hollywood',
 'gpe_Puerto Rico',
 'gpe_UK']

{INTRO AND DESCRIPTION}

Goals:

• {Evaluating the dataset and overall quality/usefulness and whether that would enable us to build a robust model:
  • { TODO: consider the garbage in garbage out approach }
  • { }
• Topic analysis

{{EXPORT HTML AND SUMMARIZE }}

Dataset Summary

The dataset is suffering from relatively extreme temporal bias. The absence of "fake news" samples before 2016 and the sudden increase after mid 2017 will likely result in any potential models overfitting on events from specific time periods.

Because of this we should be very careful about using features based on specific keywords or entities.

	Fake_Ratio	Total_Count
subject
Government News	1.0	1205
Middle-east	1.0	616
News	1.0	7343
US_News	1.0	640
left-news	1.0	3478
politics	1.0	5225
politicsNews	0.0	9080
worldnews	0.0	8251

The subject field seems to be a subset of the target variable making it completely useless for any model or analysis.

Semantic Analysis etc.

We've used Spacy and other techniques to generate a set of features based on various semantic and other attributes of each article:

Lexical Features

avg_word_length, lexical_diversity, readability, stop_words_ratio

Measure vocabulary complexity and text readability. Average word length and lexical diversity indicate vocabulary richness, while readability scores (e.g., Flesch Reading Ease) quantify overall text complexity. Stop words ratio reflects the proportion of common functional words, potentially indicating formality or information density.

Syntactic Features

pos_ratio_*, max_dep_depth

Capture grammatical structure. POS ratios show distribution of word types (nouns, verbs, etc.), revealing syntactic patterns. Max dependency depth indicates sentence complexity through nesting of grammatical structures.

Semantic Features

ner_density, avg_token_vector_norm, sentiment

Analyze meaning and content. NER density indicates fact density, average token vector norm (from spaCy's word vectors) suggests semantic richness, and sentiment score captures emotional tone.

Structural Features

avg_sent_length, median_sent_length, std_sent_length, punctuation_density

Describe text organization. Sentence length statistics reveal writing style and complexity. Punctuation density can indicate sentence structure complexity.

Temporal Features

verb_tense_ratio_*

Show time orientation through ratios of past, present, and future tenses. Useful for analyzing narrative focus in news or historical texts.

Entity Features

num_*_entities, *_entities

Identify key subjects. Entity counts and types (Person, Organization, Geopolitical Entity) reveal main text subjects. Top entities indicate key players or themes.

These are all the features that are significantly correlated (p<0.05 and > 0.1 coef) with the target variable:

	raw_correlation	abs_correlation
pos_ratio_PRON	0.45	0.45
pos_ratio_ADV	0.45	0.45
verb_tense_ratio_past	-0.45	0.45
std_sent_length	0.41	0.41
median_sent_length	-0.39	0.39
avg_sent_length	-0.32	0.32
readability	0.31	0.31
ner_density	-0.31	0.31
pos_ratio_X	0.29	0.29
avg_word_length	-0.28	0.28
stop_words_ratio	0.25	0.25
lexical_diversity	-0.20	0.20
punctuation_density	0.19	0.19
avg_token_vector_norm	-0.14	0.14
pos_ratio_PART	-0.13	0.13

We can see that "fake news" articles have a distinctive linguistic patterns compared to Reuters news reports which is not particularly surprising.

They tend to use more pronouns and adverbs, suggesting a more personal and descriptive style. These articles are less likely to use past tense, potentially focusing more on current or future events.

The sentence structure in fake news is characterized by greater variability in length, but with generally shorter sentences overall. This contrasts with legitimate news, which tends to have more consistent and slightly longer sentences.

Fake news articles often have higher readability scores, indicating simpler language, and shorter average word lengths. They also contain fewer named entities (specific people, places, or organizations), suggesting less detailed or fact-based content.

Interestingly, fake news articles are more likely to include unusual words, symbols, or potential grammatical errors. This could indicate less rigorous editing or a more casual writing style.

Again these findings only apply when comparing again articles published by Reuters specifically and not non "fake" news in general.

	raw_correlation	abs_correlation
pos_ratio_INTJ	0.36	0.36
pos_ratio_NOUN	-0.32	0.32
pos_ratio_SCONJ	0.30	0.30
pos_ratio_ADP	-0.26	0.26
pos_ratio_SPACE	0.20	0.20
pos_ratio_SYM	0.18	0.18
pos_ratio_DET	-0.18	0.18
pos_ratio_PUNCT	-0.12	0.12
pos_ratio_ADJ	-0.12	0.12
pos_ratio_AUX	0.10	0.10

If we analyze ratios of specific grammatical elements and word types inside both samples, we see some further distinctions:

• Fake news uses more interjections and subordinating conjunctions, likely indicating a more emotive style and possibly more complex sentence structures. However, they employ fewer nouns, adpositions, and determiners, suggesting less specific information.

• They also have higher ratios of spaces and symbols, with slightly more auxiliary verbs, but less punctuation and fewer adjectives. This reinforces the hypothesis of a more informal, emotionally charged style in fake news, with potentially less emphasis on concrete facts compared to Reuters articles.

Building a simple model based on semantic features

{TODO: expand}

	0	1	accuracy	macro avg	weighted avg
precision	0.972950	0.976971	0.975021	0.974960	0.975026
recall	0.975476	0.974595	0.975021	0.975035	0.975021
f1-score	0.974211	0.975781	0.975021	0.974996	0.975022
support	3466.000000	3700.000000	0.975021	7166.000000	7166.000000

We can see that we can build a relatively accurate model based on linguistic and stylometric features that were built with Spacy without actually looking at the actual content of the articles themselves.

150

Entity Analysis

Next we'll look into the actual content of the articles. We've used spacy to extract specific entities that were used in articles (specifically names of persons, geographic locations and organizations).

150 new features (50 for each category) were generated based on the most frequently referenced entities in all samples:

Most Popular Entities

We'll start with a temporal analysis of the most popular entities:

/tmp/ipykernel_5925/2659568793.py:18: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
  plot_popular.show()
/tmp/ipykernel_5925/2659568793.py:19: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
  plot_variable.show()

entity	@realDonaldTrump	Assad	Bannon	Biden	Brown	Bush	Carson	Clinton	Comey	Conway	...	Texas	Trump	Turkey	U.S.	UK	Ukraine	Virginia	Washington	Yemen	the United States
entity_type	person	person	person	person	person	person	person	person	person	person	...	gpe	gpe	gpe	gpe	gpe	gpe	gpe	gpe	gpe	gpe
date
2015-03	0.000000	0.000000	0.000000	0.055556	0.055556	0.000000	0.000000	0.166667	0.000000	0.000000	...	0.000000	0.000000	0.000000	0.200000	0.000000	0.000000	0.000000	0.600000	0.000000	0.000000
2015-04	0.000000	0.000000	0.000000	0.008333	0.054167	0.016667	0.012500	0.145833	0.000000	0.004167	...	0.030879	0.000000	0.004751	0.161520	0.009501	0.009501	0.009501	0.054632	0.007126	0.149644
2015-05	0.000000	0.000000	0.000000	0.020325	0.024390	0.028455	0.012195	0.146341	0.000000	0.000000	...	0.060526	0.000000	0.007895	0.197368	0.005263	0.002632	0.002632	0.060526	0.005263	0.089474
2015-06	0.014440	0.000000	0.000000	0.003610	0.032491	0.014440	0.003610	0.111913	0.000000	0.000000	...	0.062937	0.006993	0.003497	0.132867	0.010490	0.000000	0.010490	0.066434	0.000000	0.104895
2015-07	0.086093	0.006623	0.000000	0.000000	0.023179	0.039735	0.013245	0.052980	0.000000	0.000000	...	0.021429	0.026190	0.002381	0.166667	0.004762	0.000000	0.014286	0.050000	0.004762	0.095238
2015-08	0.082317	0.003049	0.006098	0.024390	0.033537	0.027439	0.006098	0.112805	0.000000	0.000000	...	0.054167	0.025000	0.000000	0.133333	0.008333	0.000000	0.016667	0.041667	0.000000	0.087500
2015-09	0.062500	0.031250	0.000000	0.026786	0.008929	0.013393	0.026786	0.071429	0.000000	0.004464	...	0.057143	0.014286	0.007143	0.125000	0.000000	0.000000	0.007143	0.060714	0.000000	0.089286
2015-10	0.046154	0.007692	0.000000	0.035897	0.010256	0.038462	0.023077	0.125641	0.007692	0.000000	...	0.024793	0.000000	0.002755	0.129477	0.008264	0.008264	0.016529	0.044077	0.005510	0.079890
2015-11	0.048701	0.016234	0.000000	0.003247	0.006494	0.025974	0.019481	0.084416	0.003247	0.009740	...	0.039823	0.017699	0.026549	0.130531	0.006637	0.002212	0.002212	0.053097	0.002212	0.084071
2015-12	0.062323	0.008499	0.000000	0.011331	0.002833	0.016997	0.011331	0.048159	0.011331	0.002833	...	0.008032	0.036145	0.008032	0.150602	0.006024	0.002008	0.008032	0.030120	0.008032	0.090361
2016-01	0.100234	0.000937	0.001405	0.011241	0.006557	0.043560	0.010304	0.067447	0.000000	0.000000	...	0.043162	0.040562	0.004160	0.136245	0.007800	0.000000	0.010920	0.051482	0.002080	0.113885
2016-02	0.091929	0.000362	0.000000	0.008686	0.006515	0.042345	0.022801	0.064423	0.000724	0.000362	...	0.061252	0.045372	0.002722	0.129310	0.002269	0.001815	0.015880	0.050363	0.001361	0.110254
2016-03	0.123330	0.002784	0.000000	0.003898	0.008352	0.021158	0.007795	0.065980	0.001949	0.001114	...	0.046479	0.080819	0.001734	0.136663	0.004509	0.003469	0.010753	0.054804	0.002081	0.096774
2016-04	0.112870	0.004060	0.000812	0.003654	0.006902	0.018676	0.006496	0.092570	0.002030	0.000406	...	0.050219	0.058020	0.003901	0.146758	0.000975	0.005851	0.016090	0.054120	0.002925	0.105802
2016-05	0.122956	0.000000	0.000654	0.002943	0.007521	0.016351	0.005559	0.095814	0.000981	0.000327	...	0.027764	0.087189	0.001948	0.146128	0.009255	0.001461	0.021919	0.051632	0.001461	0.116902
2016-06	0.123615	0.002424	0.000693	0.005194	0.007964	0.013504	0.001039	0.114266	0.003463	0.000000	...	0.024861	0.070724	0.003429	0.146164	0.012002	0.000857	0.011144	0.045435	0.003429	0.117874
2016-07	0.107921	0.001038	0.001038	0.005189	0.010031	0.015220	0.004151	0.124524	0.010723	0.000346	...	0.038732	0.067488	0.004108	0.142606	0.004695	0.006455	0.022887	0.047535	0.000587	0.110915
2016-08	0.116339	0.000645	0.006123	0.004834	0.004834	0.013858	0.003223	0.139542	0.003867	0.005801	...	0.014475	0.103136	0.002413	0.139928	0.003619	0.007238	0.025332	0.043426	0.001206	0.092280
2016-09	0.111041	0.003490	0.000635	0.003173	0.003807	0.014277	0.002538	0.138642	0.003173	0.006980	...	0.015648	0.085086	0.005379	0.143765	0.009291	0.006357	0.013692	0.053301	0.002934	0.105623
2016-10	0.114791	0.005976	0.000747	0.006225	0.000996	0.016434	0.001245	0.154382	0.012201	0.003735	...	0.016835	0.118807	0.005772	0.142857	0.001924	0.008658	0.019240	0.054834	0.003367	0.089466
2016-11	0.112899	0.004734	0.010651	0.001657	0.002130	0.020118	0.005207	0.104615	0.008994	0.009467	...	0.016196	0.088560	0.004135	0.141626	0.008959	0.013784	0.012750	0.058236	0.001723	0.095107
2016-12	0.123891	0.004006	0.009442	0.002289	0.008298	0.023176	0.008870	0.064664	0.004578	0.009442	...	0.017268	0.097724	0.005102	0.153846	0.004710	0.009027	0.010204	0.063579	0.003925	0.100078
2017-01	0.123987	0.003577	0.007630	0.004053	0.004530	0.017883	0.004530	0.043157	0.004769	0.008345	...	0.011667	0.093056	0.002500	0.160000	0.006389	0.007222	0.007222	0.077778	0.011944	0.105833
2017-02	0.123184	0.003196	0.021209	0.000581	0.004067	0.019175	0.003486	0.030215	0.001162	0.015398	...	0.012123	0.082785	0.004503	0.162106	0.005542	0.012470	0.011431	0.066851	0.022515	0.115691
2017-03	0.115934	0.002157	0.008088	0.001348	0.004044	0.015638	0.004583	0.028310	0.015368	0.007819	...	0.021466	0.108808	0.006662	0.165433	0.005922	0.008512	0.014064	0.061806	0.010363	0.101406
2017-04	0.120399	0.028896	0.009976	0.000688	0.000688	0.012728	0.007568	0.033712	0.005504	0.003440	...	0.013785	0.081595	0.005589	0.157973	0.002981	0.003726	0.007452	0.069300	0.004471	0.100224
2017-05	0.106183	0.002590	0.004856	0.000647	0.002590	0.012302	0.005827	0.048559	0.071544	0.008093	...	0.016997	0.098206	0.005666	0.154863	0.001889	0.004249	0.012748	0.069877	0.006610	0.091124
2017-06	0.122746	0.005694	0.003164	0.003796	0.002214	0.009807	0.004113	0.037963	0.045555	0.008225	...	0.013211	0.094476	0.002002	0.157326	0.013611	0.004003	0.022818	0.070456	0.008407	0.091673
2017-07	0.118481	0.004522	0.005125	0.001206	0.002713	0.010853	0.007235	0.056376	0.010552	0.005427	...	0.011471	0.081114	0.003277	0.157722	0.006555	0.014338	0.015158	0.058583	0.004916	0.089717
2017-08	0.138767	0.001836	0.017254	0.001836	0.003671	0.012482	0.005140	0.040749	0.005874	0.002937	...	0.023238	0.072339	0.004123	0.143178	0.004873	0.008996	0.042354	0.059220	0.004123	0.095952
2017-09	0.124822	0.017099	0.007125	0.001425	0.005985	0.010544	0.003420	0.025363	0.003990	0.000570	...	0.013745	0.024247	0.029189	0.118919	0.008340	0.009730	0.007876	0.043707	0.007104	0.094054
2017-10	0.115757	0.014124	0.003877	0.001662	0.004985	0.009970	0.002215	0.042371	0.008031	0.001385	...	0.008389	0.026907	0.026749	0.132953	0.012662	0.006173	0.007281	0.056347	0.006331	0.096233
2017-11	0.114820	0.017623	0.004539	0.005073	0.003204	0.008278	0.002403	0.042190	0.006943	0.001335	...	0.010010	0.029545	0.021311	0.128834	0.010010	0.011947	0.008072	0.043429	0.017275	0.092347
2017-12	0.129933	0.013965	0.003339	0.000607	0.001214	0.010018	0.003339	0.034305	0.010929	0.001214	...	0.005868	0.028975	0.022006	0.135155	0.012654	0.013204	0.005868	0.050614	0.012654	0.094260
2018-02	0.137931	0.000000	0.000000	0.000000	0.000000	0.000000	0.017241	0.051724	0.000000	0.000000	...	0.019231	0.076923	0.000000	0.192308	0.000000	0.000000	0.000000	0.019231	0.000000	0.096154

35 rows × 150 columns

We can see some were clear patterns, e.g. the lead up to the 2016 elections is highly visible:

• Mentions of Barrack Obama are relatively frequent initially but start decreasing signficantly at the end of this term
• Mentions of Donald Trump were growing almost continuously prior to the election, whereas Hillary Clinton start off at a very high level (being the presumed Democrat nomine for a while prior to the elections) and then more or less completely collapse after she loses to Trump.

Only Entities in "Fake News" Articles

We'll filter out Reuters articles and try to check if there are any clear topic related patterns only in fake samples.

Looking at specifically only geographic and organization entities shows some interesting patterns as well.

Summary Of Entities by Target Variable

This show the ratio of specific entity tags based on whether they were used more frequently in "Fake" vs Reuters articles.

e.g. we can see that Hillary Clinton is always referenced with her full name or just last name in Reuters articles while "fake news" ones often disrespectfully just use her first name (presumably with the intention of demeaning or mocking her)

Sentiment:

polarity metric is a measure of the overall sentiment of a given text. The value of polarity ranges from -1.0 to 1.0:

A polarity of -1.0 indicates an extremely negative sentiment.
A polarity of 1.0 indicates an extremely positive sentiment.
A polarity of 0 represents a neutral sentiment.```

There isn't a huge different in most cases but we can see that e.g. that fake news articles that mention Hillary Clinton tend to have a slightly more negative sentiment than non-fake news ones. While this is not as clear for Trump.

One signfinificant issue is that we're calculated the sentiment for the entire text, instead we should be looking only at sentences or segments surrounding specific entities.

list(cl

Embedings and Building a DNN Model

{TODO: improve section name}

A significant problem with the dataset is that it's much too easy to build a robust model while using raw data because a model would significantly overfit on:

• Articles often include the name of the news agency that published it or very specific patterns e.g. Reuters reports always start in exactly the same way.

e.g. Since almost all non-"fake" articles in this dataset are written by Reuters, and they tend to begin their articles with a very specific pattern e.g.:

Due to all the issues analyzed previously we don't believe that it's possible to build a robust model that could provide any meanginful value by using the dataset directly (or in any way...):

Non-representivie dataset

The main issue is that almost all "non-fake" news articles are published by Reuters while the "fake" come from various different sources:

	Reuters %	N Reuters articles	N non-Reuters articles
fake
0	0.997980	17296	35
1	0.014434	267	18231

This means that the legitimate news samples are highly unrrepresntive and only reflect the style of language etc. used by a single organization that gennerally only tends to publish realtivelly unbiased factual news reports with minimal analysis or opinions.

It's likely that other sources (even the ones which have published the "fake news" articles in this dataset) publish at least some "non-fake" stories/articles even though they might be significantly more biased or use a language that's much harder to discern from fake news.

Effectively this means that any model we build will only be able to classify whether an article was published by Retuers or not (i.e. the model will struggle with corecctly classifying any "non-fake" articles written by Reuters".

Temporal Biases

Additional the dataset is also highly non representive temporally:

1. the number of articles that are included is extremely unenevely distributed in the included period. Morever same applies to the target variable (i.e. most newer articles are "fake" while most older ones come from Reuters).

This means that we can't use the text without removing most specific entity names from it (e.g. persons, organization, locations etc.) e.g.:

• {the proportion of articles mentioning Hillary Clinton that were fake news varied significantly based on the 2016 eleciton cycle, same applies to many other political figures and events).

• { e.g. EXPLAIN what do I mean in more abstract way}. Fake news articles mentioning organizations like ISIS had a very high ratio of {FAKE NEWS articles} in the included period, while other groups like Hezbollah or Hamas had a much lower ratios. However this highly depends on the period the samples are coming from and it's possible that it has inverted when ISIS {left the news cycle}.

• {e.t.c}

Annomyizing text samples

Therefore we chose to focus exclusively on the style and structure of all articles instead of their specific content when preparing inputs for our DNN (BERT based) model:

Example of preprocessing pipeline:

WASHINGTON (Reuters) - Ohio Governor John Kasich, who has staked his Republican presidential campaign on the upcoming contests in the Midwest, said on Monday he would not need a lead in the delegate count before his party's convention to win the nomination, according to media reports. "In order to be the nominee, you have to have a certain number of votes," Kasich said after a town hall in Monroe, Michigan, which holds its presidential primary on Tuesday. "You've got to win. You don't just say, 'Well, I have more than anybody else, therefore I'm in,'" Kasich said, according to the Washington Post. Kasich, who is rising in some opinion polls to compete with Ted Cruz for second place in Michigan's Republican primary, has embraced the idea of a brokered convention to choose the nominee if no one gets a majority of delegates in the nominating contests for the Nov. 8 election. New York billionaire Donald Trump is at the top of the dwindling Republican pack with the highest delegate count, but victories on Saturday by Cruz, the U.S. senator from Texas, cut into his lead. Kasich trails all his rivals, including U.S. Senator Marco Rubio of Florida, in delegates won [...]

First We've started by:

• Removing all sentences that include the word Reuters since even if we remove/replace the name these article are still very easy to detect because they always start with a very specific pattern.

"In order to be the nominee, you have to have a certain number of votes," Kasich said after a town hall in Monroe, Michigan, which holds its presidential primary on Tuesday. "You've got to win. You don't just say, 'Well, I have more than anybody else, therefore I'm in,'" Kasich said, according to the Washington Post. Kasich, who is rising in some opinion polls to compete with Ted Cruz for second place in Michigan's Republican primary, has embraced the idea of a brokered convention to choose the nominee if no one gets a majority of delegates in the nominating contests for the Nov. 8 election. New York billionaire Donald Trump is at the top of the dwindling Republican pack with the highest delegate count, but victories on Saturday by Cruz, the U.S. senator from Texas, cut into his lead. Kasich trails all his rivals, including U.S. Senator Marco Rubio of Florida, in delegates won [...]

Since specific proper names (persons, organizations, regions) are very highly associated with fake news articles not because of their inherent attributes due to when the article was published we've removed all proper nouns (specifically "entities" detected by Spacy) and just replaced them with their type:

1. Each entity is replaced with it's specific tag, e.g. "Kasich" becomes [PERSON0]
2. Inside the anonymized sample the same enitity retains the same id (e.g. "Kassich is always [PERSON0], Michigan is always "[GPE2]" etc.

The preprocssesing pipeline supports other more complex modes, like assinging unique keywords or hashes (unique to each sample) to each entity so that e.g. 'PERSON{N}' etc. wouldn't be repeated in each article, however we didn't notice a signficant different in classification performance.

Example sample after full transformations:

In order to be the nominee, you have to have a certain number of votes," [PERSON0] said after a town hall in [GPE1], [GPE2], which holds its presidential primary on [DATE3]. "You've got to win. You don't just say, 'Well, I have more than anybody else, therefore I'm in,'" [PERSON0] said, according to [ORG4]. [PERSON0], who is rising in some opinion polls to compete with [PERSON5] for second place in [GPE2]'s [ORG6] primary, has embraced the idea of a brokered convention to choose the nominee if no one gets a majority of delegates in the nominating contests for the [DATE7] election. [GPE8] billionaire [PERSON9] is at the top of the dwindling [ORG6] pack with the highest delegate count, but victories on [DATE10] by [PERSON11], the [GPE12] [ORG13] from [GPE14], cut into his lead. [PERSON0] trails all his rivals, including [GPE12] [ORG13] [PERSON15] of [GPE16], in delegates won [...]

Generating and Analyzing Embeddings

We've generated emebddings using BERT (without any finetuning) for all the anonymized sample (embeddings for the raw samples and the intermediate ones (all sentences mentioning "Reuters" are removed while other entity names are retained.

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[22], line 34
     30     return tsne_results
     33 # Generate t-SNE embeddings
---> 34 tsne_2d_source = get_tsne_for_embeddings(np.stack(train_df[target_embedding_columns.raw_text.embedding].values))
     35 tsne_2d_processed = get_tsne_for_embeddings(
     36     np.stack(train_df[target_embedding_columns.processed_text.embedding].values))
     37 tsne_2d_anonymized = get_tsne_for_embeddings(
     38     np.stack(train_df[target_embedding_columns.anonymized_embeddings.embedding].values))

Cell In[22], line 25, in get_tsne_for_embeddings(embeddings, perplexity, n_iter)
     24 def get_tsne_for_embeddings(embeddings, perplexity=30, n_iter=1000):
---> 25     scaler = StandardScaler()
     26     scaled_features = scaler.fit_transform(embeddings)
     28     tsne = TSNE(n_components=2, random_state=42, perplexity=perplexity, n_iter=n_iter)

NameError: name 'StandardScaler' is not defined

Clustering

UMAP and HDBSCAN were used to cluster the embeddings (512 tokens, with remainder of the text truncated).

The specific number of UMAP components (and HDBSCAN scan parameters) were by hyperparameter tuning and targetting these specific metrics:

• Silhouette Score
• Calinski-Harabasz score
• Penalty based on proportion of noise (samples not included in any cluster)

BERT embeddings reduced with t-SNE based on target variable ("fake"/non-Reuters):

Clustering, "best" configuration with 2 final clusters:

We can see that when using emebddings generated with the source text the final optimal cluster relatively closely matches the target variable indicating that it's probably highly reliant on the first sentence beggining with "{LOCATION} Reuters.

Removing the first sentence introduced signficant amounts of noise acluster do not match target variable to such a large extent.

Interestingly stripping all entity names resulted in much cleaner cluster that more closely matches the target variable (i.e. in the "Anonymized Text" version cluster = 0 seems to be a fairly strict subset of "Fake" articles while the overal decision boundary was heavilly shifted.

Further we'll use UMAP for visualization

(note that clustering was done with 30 components while visualization is using spearately generated (with same parameters) 2 component UMAP)

UMAP seems to somewhat closely match the patterns visible with T-SNE (with clearlier "decision boundaries" due to obvious reasons)

NMI (Normalized Mutual Information) measures the amount of information shared between the clustering results and the true labels. It quantifies how well the clustering aligns with the actual categories, accounting for different numbers of clusters.

ARI (Adjusted Rand Index) assesses the similarity between the clustering and true labels, adjusting for chance agreement. It provides a measure of accuracy for the clustering compared to the ground truth, with correction for random labeling.

Both metrics help evaluate clustering performance, with higher values indicating better alignment between the clustering results and the true labels.

Let's look at each cluster in more detail to what extent do they overlap with our actual target class and could they potentially be used for actual classification.

Our goal isn't to neccesarily use this as replacement for a supervised classification model, instead we want to verify our assumption that a model that uses raw unmodified article text as inputs couldn't possibly be robust or useful because it would just overfit on specific keywoards or pattern (especially those used in Reuters news reports).

i.e. if an unsupervised "mode" like HDBSCAN ends up clustering BERT embeddings in clusters that strongly overlap with the target variable that's probably the case.

Raw Text Clusters

While the performance is relatively poor compared to the model based on style and various semantic attributes we can see that even an unsupervised model was able to more or less "notice" the patterns we've discussed.

Processed Text Clusters

Anonymized Text Clusters

While on first glance it might seem suprising that enough HDBSCAN found it somewhat easier to detected the Reuters vs non-Reuters/"fake" when all entities were removed it likely indicates that after all "noise" (with all the associated temporal biases) coming from specific entity names was removed the difference in style, semantic patterns and structure became much more noticeable.

Topic Analysis

Previously we looked at the best clustering configuration that only produced 2 clusters (based on the assumption that they are likely to match the target variable). Next we'll look into configuration tha produce a higher number of cluster and try to analyze the different topic and other attributes of included samples.

Our assumption is that these clusters represent more finegrained clusters based on topic or other attributes besides being split around the target variable.

We'll use TF-IDF combined with chi-squared to identifies key terms and overal themes in each cluster. This approach highlights terms that are both frequent and unique to each cluster

1. TF-IDF (Term Frequency-Inverse Document Frequency) identifies potentially important terms in each cluster.

2. Chi-squared analysis then determines which of these terms are statistically distinctive for each cluster.

3. Named entity recognition extracts key actors, organizations, and locations from the text.

The word cloud for each topic is based on the previously extracted entity features for each sample.

HOWEVER, instead of showing the most entities for each cluster we have selected the one which are most overreprented in that specific cluster compared to full dataset while prioritizing the most popular entities (i.e. size of each word/phrase is based on equally weighted mean of it's usage in absolute terms and how more frequently is it used compare to the full dataset so that very rare entities won't be overrepresented).

Cluster Centroid Summarization

This technique generates three key outputs for each text cluster:

Representative text: The document whose embedding is closest to the cluster's centroid (mean embedding), measured by cosine similarity. Keyword list: The most frequent terms in the cluster, excluding stop words, typically 10-20 words ranked by frequency. Cluster size: The number of documents in the cluster.

To use this technique:

Calculate the centroid (mean embedding) for each cluster. Find the document closest to each centroid using cosine similarity. Aggregate all texts in each cluster and extract frequent terms. Count the documents in each cluster.

Interpretation:

Representative text: Provides a concrete example of the cluster's content. Compare across clusters to understand dataset themes. Keywords: Highlight common topics or entities. Look for unexpected combinations or variations between clusters.

TODO:

Embeddings:

[ ] plot UMAP shift in EEmbeddings between text and anonymyzed text

	fake	DROPPED_Reuters	text	anonymized_text	anonymized_text_bert-base-uncased_used_text
0	1	False	Meryl Streep and Viola Davis weren t the only ...	[PERSON0] and [PERSON1] weren t the only ones ...	[PERSON0] and [PERSON1] weren t the only ones ...
1	1	False	Never has a movie role so perfectly described ...	Never has a movie role so perfectly described ...	never has a movie role so perfectly described ...
2	0	True	A leader of the smaller Free Democrats (FDP) a...	A leader of the smaller Free Democrats ([ORG29...	a leader of the smaller free democrats ( [ org...
3	1	False	Does this president know American history or d...	Does this president know American history or d...	Does this president know American history or d...
4	0	True	Gholamhossein Mohseni Ejei named the convicted...	[PERSON0] named the convicted man as [PERSON1]...	[PERSON0] named the convicted man as [PERSON1]...
...	...	...	...	...	...
35833	1	False	Nobody can accuse the GOP of having it all tog...	Nobody can accuse the [ORG0] of having it all ...	Nobody can accuse the [ORG0] of having it all ...
35834	0	True	Tunisia late on Sunday suspended flights from ...	[GPE0] late on [DATE1] suspended flights from ...	[GPE0] late on [DATE1] suspended flights from ...
35835	0	True	Lavrov said he had last had a telephone conver...	[ORG0] said he had last had a telephone conver...	[ORG0] said he had last had a telephone conver...
35836	0	True	Sworn in days after the overthrow of Robert Mu...	Sworn in [DATE0] after the overthrow of [PERSO...	sworn in [ date0 ] after the overthrow of [ pe...
35837	1	False	This picture is representative of the violence...	This picture is representative of the violence...	This picture is representative of the violence...

35838 rows × 5 columns

['source_text',
 'source_text_bert_embedding',
 'source_text_bert-base-uncased_used_text',
 'source_text_tokens']

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 list(df_train.columns)

NameError: name 'df_train' is not defined

0        [-0.4915247, -0.67728907, -0.120532565, -0.088...
1        [-0.2743477, -0.7278832, -0.5586739, 0.0368707...
2        [-0.9693481, -0.44558665, -0.2224515, -0.60803...
3        [-0.20548154, 0.0625999, -0.05869634, 0.270725...
4        [-1.0744206, -0.2710174, -0.25682816, -0.41904...
                               ...                        
35833    [-1.0116835, -0.6445019, 0.04759822, 0.4232899...
35834    [-0.4763119, -0.48721468, -0.49852598, -0.0282...
35835    [-0.5958654, -0.012258798, -0.07052293, 0.1296...
35836    [-0.7383414, -0.25769737, -0.7854175, -1.13049...
35837    [-0.13245764, -0.7824269, 0.25529382, 0.196255...
Name: text_bert_embedding, Length: 35838, dtype: object

1
Reducing dimensionality with UMAP...
UMAP(n_components=50, n_jobs=1, random_state=42, verbose=True)
Sun Aug 18 13:31:18 2024 Construct fuzzy simplicial set
Sun Aug 18 13:31:18 2024 Finding Nearest Neighbors
Sun Aug 18 13:31:18 2024 Building RP forest with 14 trees

/usr/local/lib/python3.10/dist-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")

Sun Aug 18 13:31:20 2024 NN descent for 15 iterations

KeyboardInterrupt

[I 2024-08-18 13:44:42,488] A new study created in memory with name: no-name-a71888b8-3fa6-4251-b298-8763847a129c

1
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'
Trial failed: Can't pickle local object 'optimize_clustering.<locals>.objective_wrapper'

Optimization completed. 0/100 trials succeeded.
No trials completed successfully. Please check your objective function and parameters.

	min_samples	alpha	silhouette_score	n_clusters	noise_proportion
149	95	1.814286	0.314928	2	0.094625
151	95	2.500000	0.314928	2	0.094625
144	95	0.100000	0.314928	2	0.094625
145	95	0.442857	0.314928	2	0.094625
146	95	0.785714	0.314928	2	0.094625
...	...	...	...	...	...
28	20	1.471429	-0.047804	3	0.547875
27	20	1.128571	-0.047804	3	0.547875
26	20	0.785714	-0.047804	3	0.547875
25	20	0.442857	-0.047804	3	0.547875
24	20	0.100000	-0.047804	3	0.547875

160 rows × 5 columns

/usr/local/lib/python3.10/dist-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")

/usr/local/lib/python3.10/dist-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")

Summary of Preprocessing pipeline:

• Raw dataset

• Strip all sentences that have Reuters in them

  • (Reuters uses a very distinct pattern to being their articles which makes them easy to detect even if all entity names are removed)

• Use Spacy to:

  • Generate various numerical features:

    • • TODO LIST AND DESCRIBE - -

  • Analyze all entity names, generated binary features based on entity tags

    • e.g. 50 most used persons, organizations and geographic locations are add as binary features (based on training dataset)

  • Generate anonymized version of that sample by replacing all entity names with unique tags e.g. [PERSON_NNNNN] (the id is unique to the document)